Skip to content

ADR-0001: Substantiate completeness and lock dependencies - #1

Merged
nicholas-ruest merged 2 commits into
mainfrom
adr/0001-substantiate-completeness-and-lock-dependencies
Jul 27, 2026
Merged

ADR-0001: Substantiate completeness and lock dependencies#1
nicholas-ruest merged 2 commits into
mainfrom
adr/0001-substantiate-completeness-and-lock-dependencies

Conversation

@nicholas-ruest

@nicholas-ruest nicholas-ruest commented Jul 27, 2026

Copy link
Copy Markdown
Member

Implements ADR-0001.

Why

This repo advertised a uniformly complete, 88%-covered, production-ready system on top of a Cargo workspace whose dependency graph had never resolved. The cause was mundane: .gitignore line 4 excluded Cargo.lock, so no lockfile ever existed, and the six LLM-Dev-Ops dependencies were declared as bare registry versions. The production-ready badge, the hand-typed coverage badge, and 13 rows of ✅ Complete were all downstream of a build that had never succeeded.

The sentinel dependency was a real supply-chain hole

sentinel = "0.1" was an unnamespaced name. Checking what it actually resolves to:

$ curl -s https://crates.io/api/v1/crates/sentinel | jq '.crate | {description, created_at, repository}'
{
  "description": "A sentinel-terminated slice library.",
  "created_at": "2015-05-21T14:08:52Z",
  "repository": "https://github.com/nils-mathieu/sentinel"
}

That is an unrelated third-party no_std FFI crate, not this platform's anomaly-detection service. A cargo build would have silently linked a stranger's code under the name sentinel. The platform's actual crate is published as llm-sentinel.

The fix removes the ambiguous sentinel key entirely rather than re-pointing it, so the bare name cannot resolve to the foreign crate again:

llm-sentinel = { git = "https://github.com/LLM-Dev-Ops/sentinel", rev = "504b0ea4..." }

All six dependencies are now pinned to an explicit git source and rev. This goes further than the benchmark-exchange precedent (Cargo.toml:97), which pins a git source but floats the branch — and which still has no lockfile of its own.

Two of the original six could never have resolved under any version:

Declared Reality
llm-cost-ops = "0.1" exists (0.1.1); now git-pinned
llm-latency-lens = "0.1" not published to crates.io at all
sentinel = "0.1" resolves to an unrelated slice library
llm-shield-core = "0.2" only 0.1.0 is published — 0.2 never existed
llm-observatory-core = "0.1" exists (0.1.1); now git-pinned
llm-config-manager = "0.1" no crate by this name exists anywhere — the real one is llm-config-core

Note: no workspace member currently references any of these, which is why they never entered the lockfile and why the missing lockfile went unnoticed. They are kept, correctly pinned, so the first member to adopt one inherits a correct source rather than a name-squatted registry match.

Build fix

crates/api-grpc/build.rs pointed tonic_build at .out_dir("src/generated"), but lib.rs pulls the modules in with tonic::include_proto!, which resolves against OUT_DIR. Generated code was written where nothing read it, and because src/generated is not committed and tonic_build does not create its out_dir, a clean checkout failed with ENOENT before protoc was ever invoked. Dropping the override fixes it.

Real build and test output

cargo build --workspace --locked (rustc 1.97.1):

error: could not compile `llm-optimizer-processor` (lib) due to 28 previous errors; 148 warnings emitted
error: could not compile `llm-optimizer-cli` (lib) due to 6 previous errors; 5 warnings emitted
BUILD_EXIT=101

cargo test --workspace --lockedWORKSPACE_TEST_EXIT=101, blocked on the same processor failure.

Per-crate, via cargo build --locked -p <crate> and cargo test --locked -p <crate>:

Crate Build Tests
types PASS 14 pass
config PASS 2 pass
analyzer PASS 1 passes
actuator PASS 1 passes
storage PASS 1 passes
api PASS 1 passes
collector PASS test targets do not compile
decision PASS test targets do not compile
integrations PASS test targets do not compile
benchmarks PASS test targets do not compile
processor FAIL (28 errors)
cli FAIL (6 errors)
api-rest FAIL (via processor)
api-grpc FAIL (via processor)
llm-optimizer FAIL (via processor)

20 tests pass out of ~2,161 #[test]/#[tokio::test] attributes. The rest are in targets that do not compile, so they have never run — which is what the 88% badge was asserting over.

Root causes:

  • processor — 14 never-type-fallback errors in Redis code (state/redis_backend.rs, state/redis_backend_enterprise.rs, storage/redis.rs, deduplication/mod.rs). Single blocker for api-rest, api-grpc, and the main binary.
  • cli — imports rand without declaring it, and uses the Formatter enum as &dyn Formatter in commands/run.rs.
  • test targets — reference types and module paths that do not exist (AnthropicConfig, CreateIssueRequest, WebhookHandler; use collector:: / use decision:: instead of the real crate names).

README changes

  • Coverage badge deleted (grep -c "img.shields.io/badge/coverage" README.md0).
  • Status badge → status-alpha-orange; two other "Production Ready" strings downgraded.
  • All 13 feature rows regraded — no row claims ✅ Complete, since none has passing tests in CI. Rows now read 🚧 / / 📋 with the blocking reason, and a Crate column so each row is checkable.
  • The second component table's hand-typed LOC and test counts (e.g. "Analyzer … 49 tests", which actually runs 1) are replaced with measured counts.
  • Marketing bullets reframed as design goals, since none is benchmarked.
  • docs/TEST_COVERAGE_REPORT.md marked a historical estimate and unlinked as evidence. Its own text already conceded "Estimated based on test scope. Run cargo tarpaulin for exact coverage."

CI changes (commit d723540)

Initially blocked — the original token lacked the workflow scope and the push was rejected. A token with that scope has since been supplied and the change is now in this PR.

--locked makes CI fail if Cargo.lock is missing or has drifted from Cargo.toml, so the committed lockfile is what actually gets built rather than a graph re-resolved per run. The tarpaulin job now emits JSON alongside the Cobertura XML, prints the measured line coverage into the job summary, and uploads the report as an artifact so a specific run's number can be audited afterwards — that measured number is what replaces the deleted hand-typed 88% badge.

-      - name: Build
-        run: cargo build --verbose --all-features
+      - name: Build (locked)
+        run: cargo build --workspace --locked --verbose --all-features

       - name: Run tests
-        run: cargo test --verbose --all-features
+        run: cargo test --workspace --locked --verbose --all-features

       - name: Run doctests
-        run: cargo test --doc --all-features
+        run: cargo test --doc --workspace --locked --all-features

       - name: Run tests with coverage
-        run: cargo tarpaulin --verbose --all-features --workspace --timeout 300 --out Xml
+        run: cargo tarpaulin --locked --verbose --all-features --workspace --timeout 300 --out Xml --out Json
+
+      - name: Publish measured coverage
+        run: |
+          COV=$(python3 -c "import json;print(f\"{json.load(open('tarpaulin-report.json'))['coverage']:.2f}\")")
+          echo "Measured line coverage: ${COV}%"
+          echo "## Coverage: ${COV}%" >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload coverage artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: coverage-report
+          path: |
+            cobertura.xml
+            tarpaulin-report.json

Expect the test job to fail on this PR. It will fail on the processor and cli compile errors documented above. That failure is the point: it is now visible, which it could not be before, when there was no lockfile and no --locked build.

Deliberately out of scope

Fixing the 34 compile errors is not attempted here. The processor errors are mechanical, but the cli's use of an enum as a trait object is a structural gap, and neither can be validated at runtime from this change. Bundling them would obscure a dependency-hygiene and claim-substantiation PR behind a large, unverifiable code change. They are documented in the README's "Known blockers" so the next change has a specific target.

ADR-0001 verification checklist

  • Cargo.lock exists at repo root and is tracked by git
  • cargo build --workspace --locked succeeds from a clean checkout — fails; that is the finding, and status is now alpha accordingly
  • cargo test --workspace executes and reports a pass count — 20 passing, per-crate; workspace-wide blocked on processor
  • No bare-version LLM-Dev-Ops dependency remains — all six carry git + rev
  • sentinel explicitly sourced, not resolved by bare name — key removed; now llm-sentinel git-pinned
  • grep -c "img.shields.io/badge/coverage" README.md returns 0
  • Every ✅ Complete row maps to a passing CI target — vacuously true; no row claims Complete
  • No production--ready badge while the above is unmet
  • CI runs cargo build --workspace --locked, cargo test, and a tarpaulin coverage job — commit d723540
  • npm run check:claims-honesty exits 0 — that script does not exist yet in agentics-enforcement; blocked on ADR-013

…l build

The workspace had never resolved its dependency graph: .gitignore excluded
Cargo.lock, so no lockfile existed and the six LLM-Dev-Ops dependencies in
[workspace.dependencies] were bare registry versions. Every README claim --
the production-ready badge, the hand-typed 88% coverage badge, and 13 rows of
"Complete" -- sat downstream of a build that had never succeeded.

Supply chain: `sentinel = "0.1"` resolved by bare name to an unrelated
crates.io crate ("a sentinel-terminated slice library", first published 2015),
not this platform's anomaly-detection service. That entry is now
`llm-sentinel`, git-pinned to LLM-Dev-Ops/sentinel. Removing the ambiguous
`sentinel` key entirely means it cannot silently resolve to the foreign crate
again. All six deps are now pinned to an explicit git source and rev. Two of
the old declarations could never have resolved at all: llm-latency-lens is not
published to crates.io, and no crate named llm-config-manager exists in any
source (the real crate is llm-config-core).

Build: crates/api-grpc/build.rs pointed tonic_build at src/generated, but
lib.rs reads the modules via tonic::include_proto!, which resolves against
OUT_DIR. The generated code was written where nothing read it, and since
src/generated is not committed and tonic_build does not create it, a clean
checkout failed with ENOENT. Dropping the out_dir override fixes it.

With the graph resolved, the real result is that the workspace does not build:
processor fails on 14 never-type-fallback errors in its Redis code, which also
blocks api-rest, api-grpc, and the main binary; cli fails on an undeclared rand
import and use of the Formatter enum as a trait object. Of ~2,161 test
attributes, 20 tests pass; the test targets in collector, decision,
integrations, and benchmarks reference types that do not exist.

README is regraded against that evidence: no row claims Complete, the coverage
badge is deleted, status is alpha, and TEST_COVERAGE_REPORT.md is marked a
historical estimate rather than linked as evidence.

The matching .github/workflows/ci.yml change -- build/test with --locked and a
tarpaulin job that publishes a measured coverage number -- is NOT in this
commit. The available token lacks the `workflow` scope, so it cannot push
workflow files. The exact diff is in the PR description and needs to be applied
by someone whose token carries that scope.

Implements auto-optimizer/docs/adr/ADR-0001-substantiate-completeness-and-lock-dependencies.md
Completes the CI half of ADR-0001, which could not be pushed with the previous
token because it lacked the `workflow` scope.

--locked makes CI fail if Cargo.lock is missing or has drifted from Cargo.toml,
so the committed lockfile is what actually gets built rather than a graph
re-resolved per run. Applied to build, test, and doctests, each scoped to
--workspace.

The tarpaulin job now emits JSON alongside the Cobertura XML and prints the
measured line coverage into the job summary. That number replaces the
hand-typed 88% badge deleted from README.md: per ADR-013 Rule 1 a coverage
figure has to come out of the run that computes it. The report is also uploaded
as an artifact so a specific run's number can be audited after the fact.

Note the test job is expected to fail until the processor and cli compile
errors documented in README.md are fixed. That failure is now visible, which it
was not before -- there was no lockfile and no --locked build.

Implements auto-optimizer/docs/adr/ADR-0001-substantiate-completeness-and-lock-dependencies.md
@nicholas-ruest
nicholas-ruest merged commit a65ffec into main Jul 27, 2026
2 of 9 checks passed
@nicholas-ruest
nicholas-ruest deleted the adr/0001-substantiate-completeness-and-lock-dependencies branch July 27, 2026 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant